1. Field of the Invention
This invention relates to a visual user interface for use in controlling the interaction of a device with a spatial region, such as the acquisition or display by a video device of a sequence of filmed scenes that are each part of a panoramic scene.
2. Related Art
A conventional video camera typically includes a viewfinder, the view through which corresponds generally to a portion (the filmed scene) of a scene that is being filmed. During conventional operation of such a video camera, the camera operator looks through the viewfinder while filming so that camera operator knows what is being filmed and so that the camera operator can control the camera to film a desired sequence of filmed scenes. However, during such operation, the camera operator""s view is limited to what can be seen through the viewfinder. Since the camera operator cannot view portions of a scene adjacent to the filmed scene, the camera operator is deprived of context for the filmed scene. This limitation impairs the camera operator""s ability to operate the camera so as to film a scene in a desired way (i.e. to acquire a desired sequence of filmed scenes). It has historically been part of the art of a skilled camera operator to be able to remember the content of a scene having a scope larger than can be encompassed within the field of view of the viewfinder of the video camera, so that the camera operator can seamlessly control the camera to change the filmed scene in a desired manner.
A video camera can he hand-held. A video camera can also be mounted on a camera support device that typically provides multiple degrees of freedom of movement of the camera. The above limitation in operation of a conventional video camera applies to both hand-held and mounted camera operation in which the camera operator controls the content of the filmed scene by looking through the viewfinder. While the camera operator can periodically look away from the viewfinder to observe the larger context of the scene being filmed, such operation is undesirable for a variety of reasons. For example, the camera operator may inadvertently misdirect the camera while not looking through the viewfinder. It is also difficult for the camera operator to readjust image perception and focus each time the operator looks away from the viewfinder to the larger scene, and vice versa. Additionally, the inability to view a current filmed scene simultaneously with the surrounding context of the current filmed scene makes it difficult for the camera operator to plan movement of the camera to film other scenes adjacent to the current filmed scene.
Mounted cameras can also be operated remotely. A video display at a remote site is connected to the video camera so that a current filmed scene is displayed by the remote video display. A control device is connected to, for example, one or more motors (or other driving devices) which are, in turn, coupled to the camera support so as to move the camera support in response to signals received from the control device. The camera operator observes the remote video display and operates the control device to control the sequence of filmed scenes acquired by the video camera. Aside from the obvious benefits (e.g., the capability of filming in areas that are hazardous to humans), remote video camera operation allows the camera operator to move away from the viewfinder so that the entire area surrounding a current filmed scene can be more easily observed during filming. However, the camera operator still cannot simultaneously view the surrounding area and the current filmed scene, so that such remote camera operation still suffers from the above-described deficiencies to an undesirable degree.
The above-described operation of a video camera is xe2x80x9cmanualxe2x80x9d operation, i.e., the camera operator must either directly or indirectly provide control of the camera at each instant in time. Automatic control of a video camera has also previously been implemented. At a predetermined location, a video camera is mounted on a camera support that allows a variety of camera motions. The video camera is connected to a control device as described above. Metrics are established for the camera support so that positions of the camera support can be correlated with particular filming directions of the video camera. During operation of the video camera, the control device, in accordance with a pre-specified instructions, transmits control signals to the motors to cause the camera support to be moved in a fashion that produces a desired sequence of filmed scenes. Using such automatic control, the position, velocity and acceleration of the video camera can be controlled at each instant in time. Automatic control of a video camera can enable dynamic control of the camera that may not be reproducible by a human operator with the same precision or speed. Automatic control can also enable pre-scripting of a sequence of filmed scenes. However, automatic control still does not satisfactorily address the above-described problems, since it does not allow real-time camera control in a manner that facilitates perception by the camera operator of the filmed scene in the context of the filmed scene""s surroundings. Filming a live event, such as a sporting event, is just one example of a situation in which such automatic camera control is inappropriate.
Once a videotape (or other set of video data) has been produced by filming a sequence of video scenes, it is desirable to have an indication of the content of the videotape and, further, an indication of where (i.e., when) particular content occurs within the videotape. A convenient way to create a record of the content of a videotape is to produce a set of time-identified annotations during filming of the videotape. However, many videotaping systems do not include a capability to produce such annotations. Creation of a handwritten set of annotations by the camera operator using, for example, paper and pencil can unduly interfere with the actual filming, thus necessitating the presence of a second person to make the annotations. Some videotaping systems include a user interface that enables selection of one of a set of predetermined textual annotations. However, such systems can be inadequate because they necessarily enable only a limited range of description of the content of the videotape. Further, these systems, while typically more convenient than using paper and pencil, may also require the presence of a second person to perform the annotation so that the camera operator is not inordinately distracted from filming. Some videotaping systems enable voice annotation to be made during filming of a videotape. However, voice annotation is inappropriate for situations in which speaking during filming of the videotape may disrupt the event or scene being filmed. Further, searching a set of audio data (here, the set of spoken annotations) for desired content can be difficult, therefore making it difficult to review a set of annotations to find a particular description of the videotape content.
In accordance with one aspect of the invention, control of the interaction of a device with a spatial region is enabled using an improved visual user interface. The visual user interface includes a display of a control space and a display of a target space. The target space typically has a specified relationship to the spatial region and, often, is a representation of some or all of the spatial region. The content of the control space and target space can also be (and often is) established such that the target space provides context for the control space. The device is operated (i.e., interacts with the spatial region) in accordance with the state of the display of the control space. For example, the device can be operated in accordance with the spatial relationship between the control space and the target space. The visual user interface according to this aspect of the invention enables a user to change the state of the display of the control space, thereby controlling the operation of the device. Generally, the display of the control space can be changed in any manner that is possible for the application with which the visual user interface is being used.
For example, a visual user interface according to this aspect of the invention can display a target scene and a control scene. Such a visual user interface can be used, for example, to control a video device. The video device controlled can be of any type, including video devices that acquire video data (e.g., video cameras, video-enabled motion picture cameras) and video devices that display video data (e.g., televisions, computer display monitors, and video or graphics projection systems). A user can, for example, cause the control scene to change in size, shape, angular orientation and/or location with respect to the target scene, which can cause, for example, a filmed scene to change in a corresponding way. Other aspects of the control scene, such as visual parameters (e.g., focus, tint, brightness), can also be changed.
A visual user interface according to this aspect of the invention can overcome, for example, the above-identified deficiencies in previous video camera operation that are associated with the limited field of view of a video camera or instance, a visual user interface according to the invention in which the target scene provides context for the control scene can facilitate perception by a video camera operator of a current filmed scene in the context of the current filmed scene""s surroundings. A camera operator""s ability to engage in critical and sophisticated camera work with great precision is greatly facilitated. The visual user interface according to this aspect of the invention can enable such difficult camera work to be performed by relatively unskilled camera operators who heretofore would not have been able to perform such camera work acceptably.
In accordance with another aspect of the invention, a visual user interface embodied in a single device enables control of the interaction of a device with a spatial region and the capability to make annotations regarding the interaction of the device with the spatial region. The visual user interface according to this aspect of the invention enables such annotations to easily be made contemporaneously with control of the device, so that a single person can perform both functions well. Additionally, the visual user interface can be implemented in a way (e.g., with a touchscreen and stylus) that enables the user to handwrite the annotations, thus facilitating a wide range of expression in creating annotations. Such a visual user interface also avoids the problems associated with voice annotation systems, e.g., disruption of the event or scene being filmed, and difficulty in searching through the annotations.
The invention can be used in a wide variety of applications. In particular, the invention is useful in applications which require the control of a video device. Examples of such applications with which the invention can be used include movie and television film production and display, security camera systems and video conferencing systems. The invention can be used, however, in applications other than those in which a video device is controlled. For example, the invention could be used in controlling a lighting device or devices so that a spatial region is illuminated in a desired way.